Sampling Techniques for Markov Games Approximation Results on Sampling Techniques for Zero-sum, Discounted Markov Games

We extend the “policy rollout” sampling technique for Markov decision processes to Markov games, and provide an approximation result guaranteeing that the resulting sampling-based policy is closer to the Nash equilibrium than the underlying base policy. This improvement is achieved with an amount of sampling that is independent of the state-space size. We base our approximation result on a more general approximation theorem, proven here, that can be used to analyze other sampling techniques to get similar guarantees. We exhibit this generality by using the theorem to provide an alternate proof of the (already known) result that the “sparse sampling” policy for Markov games approximates the Nash equilibrium. Finally, we provide empirical results showing the utility of the policy rollout technique on both a familiar “soccer game” and a difficult bandwidth-pricing domain. The latter gives an example application of the technique when the assumptions of zero-sum rewards and discounting are relaxed.